Parallel LDA Through Synchronized Communication Optimizations

Sophisticated big data machine learning applications are difficult to parallelize because it not only needs to process a big training dataset, it also needs to synchronize big model data in iterations. In parallel LDA, comparing synchronized and asynchronous communication methods under data parallelism and model parallelism, we note that the power-law distribution of word counts in LDA training datasets suggests using synchronized communication optimizations can improve the efficiency of the model update to allow the model to converge faster, shrink the model size, and further reduce the computation time in later iterations. Therefore, we abstracted new synchronized communication operations and developed two new parallel LDA implementations “lda-lgs” and “lda-rtt”. We compare our new approaches to leading implementations in the field on an Intel Haswell cluster with 100 nodes, 4000 threads. In data parallelism, “lda-lgs” can reach higher model likelihood with shorter or similar execution time compared with Yahoo! LDA. In model parallelism, when achieving similar model likelihood, “lda-rtt” can run up to 3.9 times faster compared with Petuum LDA.